Hybrid sensor system and method for providing 3D imaging

ABSTRACT

Provided is a 3D depth sensing system and method of providing an image based on a hybrid sensing array. The 3D sensing system including a light source configured to emit light, a hybrid sensing array comprising a 2D sensing region configured to detect ambient light reflected from an object and a 3D depth sensing region configured to detect the light emitted by the light source and reflected from the object, a metalens on the hybrid sensing array, the metalens being configured to direct the ambient light reflected from the object towards the 2D sensing region, and to direct the light emitted by the light source and reflected from the object towards the 3D depth sensing region, and a processing circuit configured to combine 2D image information provided by the 2D sensing region and 3D information provided by the 3D depth sensing region to generate a combined 3D image.

CROSS-REFERENCE TO RELATED APPLICATION

This application claims priority to and the benefit of U.S. ProvisionalApplication No. 63/012,000, filed on Apr. 17, 2020, entitled “A hybridphotodiode and SPAD array and sensor system to improve 3D depthinformation,” the entire content of which is incorporated by referenceherein.

BACKGROUND 1. Field

Aspects of one or more embodiments of the present disclosure relate to ahybrid sensor system and method for 3D imaging.

2. Description of the Related Art

In recent years, applications that rely on computer vision such asadvanced driver assistance systems (“ADAS”), autonomous drivingapplications, augmented reality (“AR”), virtual reality (“VR”), and thelike have become more widely demanded. Different detection systems suchas Light Detection and Ranging (“LiDAR”), cameras, and the like havebeen provided to support ADAS, autonomous driving applications, AR, VR,and the like. Such detection systems provide information to assist theapplications in understanding, modifying, and acting on information fromthe real-world.

However, it can be difficult to meet the demands of ADAS, autonomousdriving applications, AR, VR, and the like due to hardware and softwarelimitations which may adversely impact resolution, detection range,speed, and/or stability. For example, it can be challenging to providehigh resolution and high quality 3D depth information for ADAS andautonomous driving applications. As another example, low form-factor maybe desirable for AR and VR applications where longer detection range(e.g., detection range over 10 meters) and mobility of the device may beuseful.

Generally, to provide real-world information, multiple separatedetection systems may be used. However, translating the data from theseseparate detection systems into a high quality 3D image may bedifficult. For example, separate detection systems that provide aseparate set of 2D image data and a separate set of 3D depth data may beused to provide a 3D image by aligning corresponding points fromseparate sets of 2D image data and 3D depth data. This process mayconsider several factors and/or calibrations which may take substantialcomputational resources. Moreover, misalignment or inability to aligncorresponding points between 2D image data and 3D depth data mayundermine the effectiveness of the separate systems for computer visionapplications.

The above information disclosed in this Background section is forenhancement of understanding of the background of the presentdisclosure, and therefore, it may contain information that does notconstitute prior art.

SUMMARY

One or more example embodiments of the present disclosure are directedto a system and method for 3D imaging.

According to one embodiment of the present disclosure, there is provideda 3D sensing system. The 3D sensing system including a light sourceconfigured to emit light, a hybrid sensing array including a 2D sensingregion configured to detect ambient light reflected from an object and a3D depth sensing region configured to detect the light emitted by thelight source and reflected from the object, a metalens on the hybridsensing array, the metalens being configured to direct the ambient lightreflected from the object towards the 2D sensing region, and to directthe light emitted by the light source and reflected from the objecttowards the 3D depth sensing region, and a processing circuit configuredto combine 2D image information provided by the 2D sensing region and 3Dinformation provided by the 3D depth sensing region to generate acombined 3D image.

The processing circuit is further configured to determine 2D imageinformation of the object based on the 2D sensing region, determine 3Ddepth information of the object based on the 3D depth sensing region,and update the 2D image information of the object with the 3D depthinformation.

The 3D sensing system may further include a row decoder and a columnamplifier, the row decoder and the column amplifier being stacked belowthe hybrid sensing array.

The metalens may be circular, cylindrical, rectangular, or square inshape and be polarization-independent.

The 2D sensing region may include one or more photodiodes and the 3Ddepth sensing region may include one or more APDs or SPADs.

The metalens may be configured to direct NIR light to the 3D depthsensing region.

The processing circuit may be further configured to detect the objectbased on the 2D image information, generate a region of interest basedon coordinates of the object in the 2D image information, and illuminatethe region of interest using the light source.

The processing circuit may be further configured to determine anidentity of the object based on the 2D image information and 3D depthinformation, track the object based on 2D image information, and updatethe 2D image information with 3D depth information in response to acritical condition.

According to one embodiment of the present disclosure, there is provideda method of providing an image based on a hybrid sensing array. Themethod including emitting light from a light source, detecting, by a 2Dsensing region of the hybrid sensing array, ambient light reflected froman object, detecting, by a 3D depth sensing region of the hybrid sensingarray, the light emitted by the light source and reflected from theobject towards the 3D depth sensing region, directing, by a metalens onthe hybrid sensing array, ambient light towards the 2D sensing region,directing, by the metalens of the hybrid sensing array, light emitted bythe light source and reflected from the object towards the 3D depthsensing region, and combining, by a processing circuit, 2D imageinformation provided by the 2D sensing region and 3D informationprovided by the 3D depth sensing region to generate a combined 3D image.

The method may further include determining, by the processing circuit,2D image information of the object based on the 2D sensing region,determining, by the processing circuit, 3D depth information of theobject based on the 3D depth sensing region, and updating, by theprocessing circuit, the 2D image information of the object with the 3Ddepth information.

The hybrid sensing array detecting the ambient light and the lightemitted by the light source may be stacked above a row decoder and acolumn amplifier.

The metalens directing ambient light towards the 2D sensing region anddirecting light emitted by the light source and reflected from theobject towards the 3D depth sensing region may be circular, cylindrical,rectangular, or square in shape and may be polarization-independent.

The 2D sensing region may include one or more photodiodes to detectambient light and the 3D depth sensing region includes APDs or SPADs todetect the light emitted by the light source and reflected off theobject.

The directing, by the metalens on the hybrid sensing array, ambientlight towards the 2D sensing region may include directing visible lighttowards the 2D sensing region, and the directing, by the metalens of thehybrid sensing array, light emitted by the light source and reflectedfrom the object towards the 3D depth sensing region includes directingNIR light towards the 3D depth sensing region.

The method may further include detecting, by the processing circuit, theobject based on the 2D image information, generating, by the processingcircuit, a region of interest based on coordinates of the object in the2D image information, and illuminating, by the processing circuit, theregion of interest using the light source.

The method may further include determining, by the processing circuit,an identity of the object based on the 2D image information and 3D depthinformation, tracking, by the processing circuit, the object based on 2Dimage information, and updating, by the processing circuit, the 2D imageinformation with 3D depth information in response to a criticalcondition.

According to one embodiment of the present disclosure, there is provideda 3D sensing system. The 3D sensing system including a light sourceconfigured to emit light, a 2D sensing region to detect visible lightreflected from an object, a 3D depth sensing region to detect nearinfrared light reflected from the object, a metalens covering the 2Dsensing region, the metalens being configured to direct visible lightreflected from the object toward the 2D sensing region and to direct thelight emitted by the light source and reflected from the object towardsthe 3D depth sensing region, and a processing circuit configured todetermine 2D image information of the object based on the 2D sensingregion, determine 3D depth information of the object based on the 3Ddepth sensing region, and update the 2D image information of the objectwith the 3D depth information.

The 3D sensing system may further include a row decoder and a columnamplifier, the row decoder and the column amplifier being stacked belowthe 2D sensing region and the 3D depth sensing region.

The metalens may be circular, cylindrical, rectangular, or square inshape and may be polarization-independent.

The 2D sensing region may include one or more photodiodes and the 3Ddepth sensing region includes one or more SPADs or APDs.

BRIEF DESCRIPTION OF THE DRAWINGS

The above and other aspects and features of the present disclosure willbecome more apparent to those skilled in the art from the followingdetailed description of the example embodiments with reference to theaccompanying drawings.

FIG. 1 is a block diagram of a 3D image system, according to one or moreembodiments of the present disclosure.

FIG. 2A is a block diagram including a hybrid sensor, according to oneor more embodiments of the present disclosure.

FIG. 2B is a block diagram including a hybrid sensor, according to oneor more embodiments of the present disclosure.

FIGS. 3A-3I are plan views of portions of hybrid sensing arrays,according to one or more embodiments of the present disclosure.

FIGS. 4A-4D are plan views of portions of hybrid sensing arrays with aplurality of metalenses on the hybrid sensing arrays, according to oneor more embodiments of the present disclosure.

FIGS. 5A-5C are cross-sectional views of a hybrid sensing array with ametalens and a microlens on the hybrid sensing array, according to oneor more embodiments of the present disclosure.

FIG. 6A is a block diagram including a metalens and an exploded view ofa metalens on a hybrid sensing array, according to one or moreembodiments of the present disclosure.

FIG. 6B is an exploded block diagram including a lens and a metalens ona hybrid sensing array, according to one or more embodiments of thepresent disclosure.

FIG. 7A is a plan view and a cross-sectional view of a hybrid sensor,according to one or more embodiments of the present disclosure.

FIG. 7B is a cross-sectional view of a hybrid sensor, according to oneor more embodiments of the present disclosure.

FIG. 8 is a flow chart of a method for object detection and tracking,according to one or more embodiments of the present disclosure.

FIG. 9 is a flow chart of a method for providing a high resolution 2Dimage and a high resolution 3D image, according to one or moreembodiments of the present disclosure.

DETAILED DESCRIPTION

Hereinafter, example embodiments will be described in more detail withreference to the accompanying drawings, in which like reference numbersrefer to like elements throughout. The present disclosure, however, maybe embodied in various different forms, and should not be construed asbeing limited to only the illustrated embodiments herein. Rather, theseembodiments are provided as examples so that this disclosure will bethorough and complete, and will fully convey the aspects and features ofthe present disclosure to those skilled in the art. Accordingly,processes, elements, and techniques that are not necessary to thosehaving ordinary skill in the art for a complete understanding of theaspects and features of the present disclosure may not be described.Unless otherwise noted, like reference numerals denote like elementsthroughout the attached drawings and the written description, and thus,descriptions thereof may not be repeated.

Generally, LiDAR systems, for example, such as direct time-of-flight(TOF) LiDAR systems, measure a distance (e.g., a depth) of an objecttherefrom by emitting light pulses (e.g., laser pulses) toward theobject and measuring a time it takes for the light pulses to reflect offthe object and to be detected by a sensor of the LiDAR system. SomeLiDAR systems may use a 3D depth sensor including photodetectors such asa single-photon avalanche diode (SPAD) and/or an avalanche photodiode(APD) to detect photons for 3D depth information. Some LiDAR systems mayalso use a separate 2D image sensor including photodetectors such as aphotodiode (PD) to detect light at certain wavelengths (e.g., visiblewavelengths) for 2D information (e.g., a 2D image), and may fuse the 3Ddepth information with the 2D information by aligning correspondingpoints between different views of the 3D depth sensor and the 2D sensor.By fusing 3D depth information with 2D information, computer visionapplications such as object detection for autonomous drivingapplications, ADAS, and/or the like may detect and identify objects inthe real-world. Accordingly, the speed and resolution at which the fused2D image and 3D depth data is provided may be critical for real-timecomputer vision applications.

However, using separate 2D image and 3D depth sensors leads to severalchallenges. For example, the 2D image and 3D depth sensors may havedifferent fields of view (FOV), different resolutions, larger formfactor, and/or the like. Therefore, the sensors may be undesirablybulky, precise calibration settings may be demanded, and substantialcomputing power may be dedicated to aligning corresponding pointsbetween 3D depth information and 2D image data.

Depending on the FOV of the separate 2D image sensor and the FOV of theseparate 3D depth sensor, some points may not be capable ofcorresponding due to areas where the FOV do not overlap, obstructions ofone of the sensors, and/or the like. Further, the separate 2D imagesensor and the separate 3D depth sensor may have a large baselinedistance (i.e., a distance between the 2D image sensor and the 3D depthsensor) which may make alignment between 2D and 3D information moredifficult.

Individually, combining data from a separate 2D image sensor and aseparate 3D depth sensor may result in issues that may need to beaddressed during alignment. For example, a separate 3D depth sensor mayhave noise such as temporal, spatial, or flying pixels which could makefusing 2D image information and 3D depth information more difficult.

Moreover, separate 2D image and 3D depth sensors may scan all availablesensing pixels to provide data for alignment. However, this process cantake substantial computing resources and power while scanning allavailable sensing pixels may not be desired for the computer visionapplication. In the case of autonomous vehicles, increased powerconsumption may reduce available power which may reduce driving rangebased on a set amount of battery power. Further, increased computingdemands associated with scanning more sensing pixels may reduce thespeed at which calculations such as TOF may be completed.

Regardless, high resolution and high quality 3D depth information with alow form factor may be highly desirable for computer vision applicationsto detect objects in an environment at greater distances and provideenhanced mobility. In the case of ADAS and autonomous drivingapplications, increasing the distance at which an object may be detectedreduces the risk posed by the object. Therefore, it may be desirable todetect objects up to a range of up to at least 100 meters to about 200meters or more. In the case of VR and AR applications, low form factormay provide greater mobility, and high quality 3D depth information maybe desirable to identify distant objects (e.g., objects more than 10meters from the sensing system).

According to one or more embodiments of the present disclosure, acompact, monolithic hybrid sensor including a hybrid sensing array maybe provided. The hybrid sensing array may include a 2D sensing region(e.g., a complementary metal-oxide-semiconductor (“CMOS”) sensingregion) for detecting visible features and a 3D depth sensing region(e.g., a SPAD or APD sensing region) for sensing depth. The 2D sensingregion and the 3D depth sensing region of the hybrid sensing array mayshare the same or substantially the same FOV as each other. By sharing aFOV, calibration demands for the system may be reduced and alignment maybe simplified.

In one or more embodiments, one or more metalenses positioned betweenthe hybrid sensing array and external light may direct (or focus)visible light to the 2D sensing region and direct (or focus) nearinfrared (“NIR”) light to the 3D depth sensing region to enhance visiblelight and NIR detection.

Further, a first plurality of sensing pixels including PDs of the 2Dsensing region and a second plurality of sensing pixels includingsensing pixels of the 3D depth sensing region may be arranged such thata high resolution 2D image updated or augmented with high quality 3Ddepth information according to a shared FOV may be provided. In otherwords, a combined 3D image may be provided. For example, in one or moreembodiments, the 2D image data from the 2D sensing region may be used bya processing circuit to detect objects in a high resolution 2D viewthereby allowing object detection at a greater distance. The processingcircuit may update or argument the high resolution 2D image data with 3Ddepth information by mapping 3D depth information from the 3D depthsensing region onto the 2D image data.

In one or more embodiments of the present disclosure, the processingcircuit may conserve power and reduce computational strain by applying aregion of interest (“ROI”) scanning mechanism to the 3D depth sensingregion of a hybrid sensor. For example, an object detector of theprocessing circuit may use an algorithm to identify and track objects ofinterest (“OOI”) based on a 2D image. Based on the identified andtracked OOI, in one or more embodiments, a ROI controller of theprocessing circuit may communicate with the sensing circuit such that aportion of the sensing pixels of the 3D depth sensing region (e.g., aportion of the SPADs or APDs of the 3D depth sensing region) provide 3Ddepth information to the processing circuit instead of all of thesensing pixels of the 3D depth sensing region. Further, the ROIcontroller of the processing circuit may, in one or more embodiments,communicate with a light source, directly or indirectly, to activate aportion of an array of vertical-cavity surface-emitting lasers(“VCSELs”)) to illuminate the tracked OOI. In the case of ADAS andautonomous vehicle applications, for example, a pedestrian may be mobilein contrast to a tree which may be static, and therefore, the 3D depthinformation of the pedestrian may be provided using ROI scanning while3D depth information of the tree may not be collected to save powerand/or computational resources.

The above and other aspects and features of one or more exampleembodiments of the present disclosure will now be described in moredetail with reference to the figures.

FIG. 1 is a block diagram 100 of a 3D sensing system 102, according toone or more embodiments of the present disclosure.

Referring to FIG. 1, according to one or more example embodiments of thepresent disclosure, a 3D sensing system 102 may acquire informationcorresponding to its surroundings in an environment. In one or moreembodiments, the 3D sensing system includes a light source 104 (e.g., anarray of VCSELs) and a hybrid sensor 106.

The hybrid sensor 106 may include a 2D sensing region configured togenerate 2D information (e.g., 2D image information with or withoutcolor) based on ambient light and a 3D depth sensing region configuredto generate 3D depth information according to TOF measurements based onlight emitted from the light source 104.

For example, the 2D sensing region of the hybrid sensor 106 may detectambient light reflected from one or more targets to generate 2D imageinformation and the 3D depth sensing region of the hybrid sensor 106 maydetect light emitted (e.g., a light pulse emitted) from the light source104 and reflected back towards the hybrid sensor 106 (which may includeone or more metalenses to direct the reflected light towards the 3Ddepth sensing region). According to the time elapsed (i.e., TOF) fromemission of a light pulse from the light source 104 to detection of thereflected light pulse by the 3D depth sensing region, the 3D sensingsystem 102 may determine the distance D1 (e.g., the depth) to thesurface of one or more objects 108.

In one or more embodiments, the 2D information may be independently usedto detect and track one or more objects 108 located within a suitableproximity to the 3D sensing system 102 at high resolution. In one ormore embodiments, the 3D depth information may be used to update the 2Dinformation of the one or more tracked objects 108 to provide ahigh-resolution 2D image supplemented or augmented with 3D depthinformation.

Accordingly, the 3D sensing system 102 may identify and track one ormore objects 108 in an environment at high resolution in 2D using thehybrid sensor 106, and may augment or update 2D informationcorresponding to the one or more objects with 3D depth information usingthe hybrid sensor 106. By detecting the one or more objects 108 at highresolution in 2D, the 3D sensing system 102 may increase objectiondetection distance and may improve depth estimation for the detectedobject.

FIG. 2A is a block diagram including a hybrid sensor 106, according toone or more embodiments of the present disclosure. FIG. 2B is a blockdiagram including a hybrid sensor, according to one or more embodimentsof the present disclosure.

Referring to FIGS. 2A-2B, according to one or more example embodimentsof the present disclosure, a hybrid sensor 106 includes a sensingcircuit 202 for detecting light and a processing circuit 204 including aprocessor 206 and memory 208. The processor 206 may be implemented as ageneral purpose processor, an application specific integrated circuit(ASIC), one or more field programmable gate arrays (FPGAs), a group ofprocessing components, or other suitable electronic processingcomponents. The memory 208 (e.g., memory, memory unit, storage device,and/or the like) may include one or more devices (e.g., RAM, ROM, Flashmemory, hard disk storage, and/or the like) for storing data and/orcomputer code for completing or facilitating the various processesdescribed in the present application. The memory 208 may be or includevolatile memory or non-volatile memory. The memory 208 may includedatabase components, object code components, script components, or anyother type of information structure for supporting the variousactivities and information structures described in the presentapplication. According to an example embodiment, the memory 208 may becommunicably connected to the processor 206 via the processing circuit204, and includes computer code for executing (e.g., by the processingcircuit 204 and/or the processor 206) one or more processes describedherein.

As shown in FIGS. 2A-2B, the processing circuit 204 may be implementedwithin the hybrid sensor 106 as an internal processing circuit 204 ofthe hybrid sensor 106. However, the present disclosure is not limitedthereto. In one or more embodiments, the functions of the processingcircuit 204 may be separated or shared across a plurality of localprocessing circuits (e.g., the processing circuit 204 of the hybridsensor 106 and another processing circuit of the 3D sensing system 102which may be separate from the hybrid sensor 106). In one or moreembodiments, the processing circuit of the 3D sensing system 102 may,for example, control functions of the 3D sensing system 102 (e.g.,control functions of a motor vehicle) and communicate with theprocessing circuit 204 of the hybrid sensor 106. Therefore, in one ormore embodiments, the processing circuit 204 of the hybrid sensor 106may offload some functions that may be performed by the processingcircuit of the 3D sensing system 102 thereby freeing the processingcircuit of the 3D sensing system 102 to perform other functions.

Although one or more local processing circuits are described asperforming the functions described in the present disclosure, thepresent disclosure is not limited thereto. For example, in otherembodiments, the processing circuit 204 or one or more componentsthereof (e.g., components executing instructions in memory 208 toperform the methods described in the present disclosure) may bedistributed across multiple servers or computers (e.g., that can existin distributed locations).

In one or more embodiments, the processing circuit 204 may executeinstructions in memory 208 to function as an image signal processor 214,an object detector 212, and/or a ROI controller 210.

The image signal processor 214 may provide or reconstruct 2D imagesand/or 3D images using data from the sensing circuit 202. In one or moreembodiments, the image signal processor 214 provides or reconstructs 2Dimages by interpolating missing 2D image sensing data. The image signalprocessor 214 may provide or reconstruct 3D images by interpolatingmissing 3D depth sensing data and applying 3D depth information to theprovided or reconstructed 2D image according to an algorithm to providea high resolution 2D image supplemented, assisted, or updated with 3Ddepth information (i.e., a 3D image).

In one or more embodiments, the image signal processor 214 may improve3D quality using the 2D image. For example, the image signal processor214 may detect one or more OOI using the high-resolution 2D image, andmay direct 3D depth sensing towards a region of interest (ROI) where theOOI may be detected. In this case, the 2D image may be used tosupplement and/or verify the depth information of the 3D image, forexample, such as filling missing 3D information of the OOI consistentwith the 2D information.

The object detector 212 may use an algorithm including a neural-networkbased object detection system to detect and track objects in the 2Dimage data based on data from the 2D sensing region. In one or moreembodiments, the object detector 212 may track objects in the 2D imageand update specifically the regions (e.g., the ROI) of the trackedobjects with 3D information (e.g., 3D depth information) based on ROIscanning.

The ROI controller 210 may use object detector 212 tracking informationto interact with the sensing circuit 202 and/or a light source 104 toimplement ROI based 3D sensing. For example, the ROI controller 210 maytransmit signals to the sensing circuit 202 such that the row decoder218 and/or the column scanner 220 perform either normal or ROI basedscans according to row and column control logic. In one or moreembodiments, the ROI controller 210 may use tracking information fromthe object detector 212 to direct a light source to illuminatespecifically one or more OOI. For example, the ROI controller maytransmit instructions to activate only a portion of the array of VCSELsof the light source directed toward the one or more tracked OOI.Therefore, battery power consumption may be reduced.

Accordingly, the image signal processor 214, object detector 212, andthe ROI controller 210 enable high resolution, high quality 3D imagesincluding object identification and tracking with ROI-based scanning oftracked objects.

Referring to FIG. 2A, in one or more embodiments, the sensing circuit202 of the hybrid sensor 106 includes a hybrid sensing array 216including a 2D sensing region and a 3D depth sensing region. The 2Dsensing region includes a first plurality of sensing pixels arrangedinto an array of lines forming columns and rows, and the 3D depthsensing region includes a second plurality of sensing pixels arrangedinto an array of lines forming columns and rows. The rows may extend ina first direction DR1 and the columns may extend in a second directionDR2 crossing the first direction DR1.

Each of the first plurality of sensing pixels may include a PD and eachof the second plurality of sensing pixels may include a SPAD or APD. Inone or more embodiments, each of the PDs of the 2D sensing region maydetect red light, green light, or blue light, and each of the SPADs orAPDs may detect near infrared light (i.e., light having a wavelengthbetween 0.8 micrometers and about 1 micrometers, such as 904 nanometersor 940 nanometers). However, the present disclosure is not limitedthereto. For example, one or more of the PDs of the 2D sensing regionmay detect any suitable colors of light in the visible spectrum to NIRspectrum (e.g., about 0.4 to about 0.8 micrometer), and one or more ofthe sensing pixels of the 3D depth sensing region may detect anysuitable spectrum of light (e.g., wavelengths of light greater than 0.8micrometers, such as 904 nanometers or 940 nanometers) for measuring TOFbased on the light source 104. In the case of the 2D sensing region, anysuitable arrangement may be used, for example, the 2D sensing region mayinclude one or more of red, green, and blue sensing pixels, one or moreof red and clear sensing pixels, one or more clear sensing pixels, orone or more monochrome sensing pixels.

In one or more embodiments, the sensing pixels of the 2D sensing regionand the sensing pixels of the 3D depth sensing region may be interwoven(e.g., interwoven in a checked arrangement as shown in the embodimentsof FIGS. 3A-3D or interwoven in a striped arrangement as shown in theembodiments of FIGS. 3E-3H) as shown and described in more detail withreference to FIG. 3A-3H below. In other words, each of the sensingpixels of the 3D depth sensing region may be directly adjacent to one ormore sensing pixels of the 2D sensing region such that a sensing pixelof the 2D sensing region may be arranged between sensing pixels of the3D depth sensing region in the first direction DR1 or the seconddirection DR2, or a sensing pixel of the 3D depth sensing region may bearranged between two sensing pixels of the 2D sensing region in thefirst direction DR1 or the second direction DR2. However, the presentdisclosure is not limited thereto. For example, the 2D sensing regionand the 3D depth sensing region may be in a side-by-side arrangement asshown in the embodiments of FIGS. 2B, 3I, 6B, and 7A-7B.

Referring to FIG. 2B, in one or more embodiments, the sensing circuit202 of the hybrid sensor 106 includes a hybrid sensing array including a2D sensing region 230 and a 3D depth sensing region 232 in aside-by-side arrangement. The 2D sensing region 230 includes a firstplurality of sensing pixels arranged into an array of lines formingcolumns and rows, and the 3D depth sensing region 232 includes a secondplurality of sensing pixels arranged into an array of lines formingcolumns and rows. The rows may extend in a first direction DR1 and thecolumns may extend in a second direction DR2 crossing the firstdirection DR1.

Each of the first plurality of sensing pixels may include a PD and eachof the second plurality of sensing pixels may include a SPAD or APD. Inone or more embodiments, each of the PDs of the 2D sensing region 230may detect red light, green light, or blue light, and each of the SPADsor APDs may detect near infrared light. However, the present disclosureis not limited thereto. For example, one or more of the PDs of the 2Dsensing region 230 may detect any suitable colors of light in thevisible spectrum to NIR spectrum (e.g., about 0.4 to about 0.8micrometer), and one or more of the sensing pixels of the 3D depthsensing region 232 may detect any suitable spectrum of light (e.g.,wavelengths of light greater than 0.8 micrometers, such as 904nanometers or 940 nanometers) for measuring TOF based on the lightsource 104. In the case of the 2D sensing region 230, any suitablearrangement may be used, for example, the 2D sensing region 230 mayinclude one or more of red, green, and blue sensing pixels, one or moreof red and clear sensing pixels, one or more clear sensing pixels, orone or more monochrome sensing pixels.

In one or more embodiments, one or more metalenses may be positioned onor above the hybrid sensing arrays in the embodiments of FIGS. 2A-2Bsuch that light incident on the one or more metalenses may be directedto different sensing regions according to the wavelength of the incidentlight. For example, light in the visible spectrum (e.g., red light,green light, blue light, and the like) may be directed (or focused) bynanostructures of the one or more metalenses to a sensing pixel of the2D sensing region while light in the NIR spectrum may be directed (orfocused) towards a sensing pixel of the 3D depth sensing region.Accordingly, the one or more metalenses improve the ability of thesensing pixels of the hybrid sensing array to detect inbound light.

Referring to FIG. 2A, in one or more embodiments, a row decoder 218 anda column scanner 220 may be used to address sensing pixels of the hybridsensor 106. The row decoder 218 and/or the column scanner 220 mayreceive and output driving signals such that select sensing pixels maydetect incident light and may output a sensing signal in response to thedetected light. Sensing signals from sensing pixels in a pixel row maybe stored in a column memory 228 prior to being read out sequentiallyto, for example, the processing circuit 204. In one or more embodiments,the sensing circuit 202 may provide signals to the row decoder 218and/or column scanner 220 in accordance with signals from the ROIcontroller 210 of the processing circuit 204 to perform normal (e.g.,general scanning) or ROI based 3D scanning.

In one or more embodiments, the sensing circuit 202 includes a columnamplifier 224 for amplifying signal, a column analog to digitalconverter (“ADC”) 226, and a correlated double sampling (“CDS”) circuit222. The column amplifier amplifies from signal of the sensing pixel tothe ADC with CDS to provide a readout signal to the processing circuit204. In one or more embodiments, the circuit providing the readoutsignal and/or the row and column control logic may be shared between the2D sensing region and the 3D depth sensing region. However, the presentdisclosure is not limited thereto. For example, the readout signaland/or the row and column control logic may be supplied by two separatecircuits connected to the 2D sensing region and the 3D depth sensingregion respectively as shown in FIG. 2B.

Referring to FIG. 2B, the 2D sensing region 230 may be connected to afirst circuit including a row decoder 234 and column circuitry 236including a column amplifier with CDS function, column ADC with digitaldouble sampling (“DDS”) function, and line-memory block. The 3D depthsensing region 232 may be connected to a second circuit including a rowdecoder 238 and column circuitry 240 including a column amplifier if itis needed, time to digital converter (“TDC”), and line-memory block. Inone or more embodiments, the first circuit and the second circuitcommunicate with or include a phase-locked loop (“PLL”), counter thatmay be gray counter or binary counter, ramp generator, and digitallogics (e.g., digital logics for reconstructing 2D and 3D images) of thesensing circuit 202 to provide 2D image data and 3D depth information.Accordingly, as shown in FIG. 2B, in one or more embodiments, the firstcircuit and the second circuit may be separate from each other such thatseparate readout and separate row and column control logics areprovided.

Referring to FIGS. 2A-2B, in one or more embodiments, the hybrid sensingarray and other blocks such as, for example, the row decoder, CDScircuit, column amplifier, the column ADC, the column memory, the columnscanner, line-memory, TDC, PLL, counter, ramp generator, digital logics,the processing circuit 204, components of the processing circuit 204,and/or the like may be implemented in the same package (e.g., the samedie) and/or may be part of a stacked structure (e.g., a stacking die).

In one or more embodiments, the hybrid sensing array and the otherblocks may be side-by-side in the same package (e.g., the same die). Inother words, the hybrid sensing array and the other blocks may notoverlap with each other in a plan view and may be connected usingconductive components (e.g., conductive traces and/or the like).

In one or more embodiments, the hybrid sensing array and the otherblocks may be part of a stacked structure where the one or more of theother blocks may be below the hybrid sensing array. The other blocks maybe connected to the hybrid sensing array and/or each other usingconductive components (e.g., hybrid bonding, through silicon-via(“TSV”), and/or the like).

FIGS. 3A-3I are plan views of portions of hybrid sensing arrays 300 a,300 b, 300 c, 300 d, 300 e, 300 f, 300 g, 300 h, 300 i according to oneor more embodiments of the present disclosure.

Referring to FIGS. 3A-3D, the 2D sensing region includes a plurality ofsensing pixels (e.g., one or more green sensing pixels 301 configured todetect a green color, one or more red sensing pixels 302 configured todetect a red color, and one or more blue sensing pixels 303 configuredto detect a blue color). In the illustrated embodiments, the 3D depthsensing region includes one or more sensing pixels 304 configured todetect NIR light. The sensing pixels 301, 302, 303 of the 2D sensingregion and the sensing pixels 304 of the 3D depth sensing region may beinterwoven in various suitable configurations. In one or moreembodiments, the 2D sensing region includes a first sensing pixel row305 with alternating green sensing pixels 301 and red sensing pixels302, and a second sensing pixel row 307 including alternating bluesensing pixels 303 and green sensing pixels 301. The first sensing pixelrow 305 may be directly adjacent to the second sensing pixel row 307 toform 2×2 square segments including two green sensing pixels 301, one redsensing pixel 302, and one blue sensing pixel 303. The 2×2 segments maybe repeated in the row direction (e.g., the first direction DR1) and thecolumn direction (e.g., the second direction DR2 crossing the firstdirection DR1) to form a Bayer image. However, the present disclosure isnot limited hereto. For example, the 2D sensing region may includesensing pixels arranged in any suitable shape with any suitablearrangement of color sensing pixels to fit other image schemes (e.g., amonochrome, red and clear, or clear image scheme).

According to one or more embodiments of the present disclosure, a ratiobetween sensing pixels of the 3D depth sensing region to a sensing pixelof the 2D sensing region may suitably vary. For example, in one or moreembodiments, the ratio between a sensing pixel of the 3D depth sensingregion to a sensing pixel of the 2D sensing region may be 1:1 (e.g., onesensing pixel including a SPAD to one monochrome sensing pixel), and inother embodiments, the ratio between a sensing pixel of the 3D depthsensing region to a sensing pixel of the 2D sensing region may be morethan 1:1 (e.g., one sensing pixel 304 per group of red, green, and bluesensing pixels 301, 302, 303 as shown in FIGS. 3A-3D). The ratio ofsensing pixels 304 of the 3D depth sensing region to sensing pixels 301,302, 303 of the 2D sensing region may affect the resolution ofcorresponding data in an interwoven arrangement as shown in FIGS. 3A-3H.For example, increasing the number of sensing pixels 301, 302, 303 ofthe 2D sensing region per sensing pixels 304 of the 3D depth sensingregion improves the resolution of the 2D sensing data while decreasingthe resolution of 3D depth sensing data. Referring to FIGS. 3A-3Ddifferent ratios of sensing pixels 304 of the 3D depth sensing region tosensing pixels 301, 302, 303 of the 2D sensing region may be provided tobalance a high resolution 2D image and minimum resolution 3D depthinformation that make possible to detect minimum size of object.

In one or more embodiments of the present disclosure, a ratio of an areaoccupied by a sensing pixel 304 of the 3D depth sensing region to anarea occupied by a sensing pixel 301, 302, 303 of the 2D sensing regionmay be 1:4 as shown in the embodiments of FIGS. 3A-3D. However, thepresent disclosure is not limited thereto. For example, the ratio of thearea occupied by a sensing pixel 304 of the 3D depth sensing region tothe ratio of the area occupied by a sensing pixel 301, 302, 303 of the2D sensing region may suitably vary (e.g., an area ratio of 1:1 or anarea ratio greater than 1:1).

As shown in FIG. 3A, sensing pixels 301, 302, 303 of the 2D sensingregion may be interrupted (or displaced) by sensing pixels 304 of the 3Ddepth sensing region such that the 2D sensing region and the 3D depthsensing region may be interwoven. In other words, a sensing pixel 304 ofthe 3D depth sensing region may be present instead of one or moresensing pixels 301, 302, 303 of the 2D sensing region according to apattern of the sensing pixel column and/or sensing pixel row of the 2Dsensing region. In one or more embodiments, the sensing pixels 304 ofthe 3D depth sensing region may be repeated at a first interval 306 inin a row direction (e.g., first direction DR1) and a second interval 308in a column direction (e.g., second direction DR2). In this case, thefirst interval 306 and the second interval 308 may be equal to eachother, and therefore, the sensing pixels 304 of the 3D depth sensingregion may have a square-shape arrangement. In the embodiment shown inFIG. 3A, the ratio of sensing pixels 304 of the 3D depth sensing regionto sensing pixels 301, 302, 303 of the 2D sensing region may be 1:12 andeach sensing pixels 304 of the 3D depth sensing region may be separatedfrom another sensing pixel 304 of the 3D depth sensing region by two ofsensing pixels 301, 302, 303 of the 2D sensing region. However, thepresent disclosure is not limited thereto. For example, the sensingpixels 304 of the 3D depth sensing region may be separated from eachother by any suitable number of sensing pixels 301, 302, 303 of the 2Dsensing region with a corresponding effect on the ratio of sensingpixels 304 of the 3D depth sensing region to sensing pixels 301, 302,303 of the 2D sensing region.

As shown in FIG. 3B, in one or more embodiments, the sensing pixels 304of the 3D depth sensing region may be repeated for a row direction(e.g., first direction DR1) and a column direction (e.g., seconddirection DR2) at a third interval 310 in in a row direction (e.g.,first direction DR1) and a fourth interval 312 in a column direction(e.g., second direction DR2). In this case, the third interval 310 andthe fourth interval 312 may be different from each other, and therefore,the sensing pixels 304 of the 3D depth sensing region may have arectangular-shape arrangement. In the embodiment shown in FIG. 3B, theratio of sensing pixels 304 of the 3D depth sensing region to sensingpixels of the 2D sensing region may be 1:20 and each of the sensingpixels 304 of the 3D depth sensing region may be separated from anothersensing pixel 304 of the 3D depth sensing region by two of sensingpixels 301, 302, 303 of the 2D sensing region in the row direction(e.g., the first direction DR1) and four of sensing pixels 301, 302, 303of the 2D sensing region in the column direction (e.g., the seconddirection DR2). However, the present disclosure is not limited thereto.For example, the sensing pixels 304 of the 3D depth sensing region maybe separated from each other by any suitable number of sensing pixels301, 302, 303 of the 2D sensing region in the row direction (e.g., thefirst direction DR1) and/or the column direction (e.g., the seconddirection DR2) with a corresponding effect on the ratio of sensingpixels 304 of the 3D depth sensing region to sensing pixels 301, 302,303 of the 2D sensing region.

As shown in FIG. 3C, in one or more embodiments, the sensing pixels 304of odd-numbered sensing pixel rows 315 of the 3D depth sensing regionmay have a first rectangular-shape arrangement (indicated by the fifthinterval 314 being different from the sixth interval 316) and thesensing pixels 304 of even-numbered sensing pixels rows 317 of the 3Ddepth sensing region may have a second rectangular-shape arrangement(matching the first rectangular-shape arrangement). The firstrectangular-shape arrangement may be offset from the secondrectangular-shape arrangement in the row direction (e.g., firstdirection DR1) and/or the column direction (e.g., second direction DR2).In other words, the sensing pixels 304 of the 3D depth sensing regionmay have a zig-zag-shape arrangement. In the embodiment shown in FIG.3C, each sensing pixel column of the 2D sensing region may beinterrupted (or displaced) by a sensing pixel 304 of the 3D depthsensing region. Further, the ratio of sensing pixels 304 of the 3D depthsensing region to sensing pixels 301, 302, 303 of the 2D sensing regionmay be 1:12 and each sensing pixel of the 3D depth sensing region may beseparated from another sensing pixel of the 3D depth sensing region bytwo of sensing pixels 301, 302, 303 of the 2D sensing region in the rowdirection (e.g., first direction DR1) and six of sensing pixels 301,302, 303 of the 2D sensing region in the column direction (e.g., seconddirection DR2). However, the present disclosure is not limited thereto.For example, the sensing pixels 304 of the 3D depth sensing region maybe separated from each other by any suitable number of sensing pixels301, 302, 303 of the 2D depth region in the row direction (e.g., thefirst direction DR1) and/or the column direction (e.g., the seconddirection DR2) with a corresponding effect on the ratio of sensingpixels 304 of the 3D depth sensing region to sensing pixels 301, 302,303 of the 2D sensing region.

As shown in FIG. 3D, in one or more embodiments, the sensing pixels 304of odd-numbered sensing pixel rows 315 of the 3D depth sensing regionmay have a first square-shape arrangement (indicated by the seventhinterval 318 being different from the eighth interval 320) and thesensing pixels 304 of even-numbered sensing pixels rows 317 of the 3Ddepth sensing region may have a second square-shape arrangement(matching the first square-shape arrangement). The first square-shapearrangement may be offset from the second square-shape arrangement inthe row direction (e.g., first direction DR1) and/or the columndirection (e.g., second direction DR2) at the same interval (i.e., theninth interval 322 is equal to the tenth interval 324). In other words,the sensing pixels 304 of the 3D depth sensing region may have adiamond-shape arrangement. In the embodiment shown in FIG. 3D, somesensing pixel columns of the 2D sensing region may not be interrupted(or displaced) by a sensing pixel 304 of the 3D depth sensing region.Further, the ratio of sensing pixels 304 of the 3D depth sensing regionto sensing pixels 301, 302, 303 of the 2D sensing region may be 1:28 andeach sensing pixels 304 of the 3D depth sensing region may be separatedfrom another sensing pixel 304 of the 3D depth sensing region by six ofsensing pixels 301, 302, 303 of the 2D sensing region in the rowdirection (e.g., first direction DR1) and six of sensing pixels 301,302, 303 of the 2D sensing region in the column direction (e.g., seconddirection DR2). However, the present disclosure is not limited thereto.For example, the sensing pixels 304 of the 3D depth sensing region maybe separated from each other by any suitable number of sensing pixels301, 302, 303 of the 2D sensing region in the row direction (e.g., thefirst direction DR1) and/or the column direction (e.g., the seconddirection DR2) with a corresponding effect on the ratio of sensingpixels 304 of the 3D depth sensing region to sensing pixels 301, 302,303 of the 2D sensing region.

Referring to FIGS. 3A-3D, each of the various arrangements may affectobject detection and tracking by the object detector 212 of theprocessing circuit 204. For example, the minimum bounding box size foridentifying an object (e.g., a human, car, obstacle, and the like) in a2D image and a 3D image may be affected by placement and ratio ofsensing pixels 304 of the 3D depth sensing region to sensing pixels 301,302, 303 of the 2D sensing region. FIG. 3A may have a minimum boundingbox size of 30×8, FIG. 3B may have a minimum bounding box size of 30×8,FIG. 3C may have a minimum bounding box size of 30×6, and FIG. 3D mayhave a minimum bounding box size of 30×8.

Referring to FIGS. 3E-3H, the sensing pixel rows 326, 328, 330, 332,334, 336 of the hybrid sensing arrays 300 e, 300 f, 300 g, 300 h may beeither sensing pixel rows 328, 330, 334, 336 of the 2D sensing region orsensing pixel rows 326, 332 of the 3D depth sensing region. In otherwords, the 2D sensing region and the 3D depth sensing region may bearranged as alternating horizontal bars in a striped arrangement.However, the present disclosure is not limited thereto. For example, thesensing pixel rows 328, 330, 334, 336 of the 2D sensing region and thesensing pixel rows 326, 332 of the 3D depth sensing region may bearranged as alternating vertical bars in the striped arrangementaccording to an application thereof. For example, in autonomous drivingapplications, objects located in front of the vehicle may be moreimportant than objects located at a side of the vehicle. Thus, in thiscase, the sensing pixel rows 326, 332 of the 3D depth sensing region maybe arranged as alternating horizontal bars in the striped arrangement.

As shown in FIG. 3E, in one or more embodiments, the sensing pixels 304of the 3D depth sensing region may be repeated consecutively in a rowdirection (e.g., a first direction DR1) and the sensing pixel 301, 302,303 of the 2D sensing region may also repeated consecutively in a rowdirection (e.g., a first direction DR1). In the illustrated embodiment,the sensing pixels 304 of the 3D depth sensing region have asquare-shape. However, the present disclosure is not limited thereto.For example, the sensing pixels 304 of the 3D depth sensing region mayhave any suitable shape such as a rectangular shape as shown in theembodiments of FIGS. 3F-3H.

Referring to FIG. 3E, in a column direction (e.g., the second directionDR2), a sensing pixel row 326 of the 3D depth sensing region may beseparated from another sensing pixel row 332 of the 3D depth sensingregion by two sensing pixel rows 328, 330 of the 2D sensing region.However, the present disclosure is not limited thereto. For example, thesensing pixel rows 326, 332 of the 3D depth sensing region may beseparated from each other by any number of sensing pixel rows of the 2Dsensing region at regular or irregular intervals as shown in FIGS. 3Gand 3H.

Referring to FIG. 3I, a hybrid sensing array 300 i may include a 2Dsensing region 338 and a 3D depth sensing region 340 separate from eachother in a side-by-side arrangement.

As shown in FIG. 3I, the 2D sensing region 338 is adjacent to the 3Ddepth sensing region 340 as a whole. In other words, the sensing pixels304 of the 3D depth sensing region 340 may be directly adjacent to eachother such that no sensing pixels 301, 302, 303 of the 2D sensing region338 may be between the sensing pixels 304 of the 3D depth sensing region340 in a first direction DR1 or the second direction DR2, and sensingpixels 301, 302, 303 of the 2D sensing region 338 may be directlyadjacent to each other such that no sensing pixels 304 of the 3D depthsensing region 340 may be between the sensing pixels 301, 302, 303 ofthe 2D sensing region 338 in the first direction DR1 or the seconddirection DR2. The side-by-side arrangement for the 2D sensing region338 and the 3D depth sensing region 340 may provide enhanced 2D imageresolution due to an increase in the density of sensing pixels 301, 302,303 of the 2D sensing region compared to an arrangement where thesensing pixel 301, 302, 303 of the 2D sensing region may be interwovenwith sensing pixels 304 of the 3D depth sensing region as shown in theembodiments of FIGS. 3A-3H.

FIGS. 4A-4D are plan views of portions of hybrid sensing arrays 400 a,400 b, 400 c, 400 d with a plurality of metalenses (e.g., metalens 406)on the hybrid sensing arrays 400 a, 400 b, 400 c, 400 d, according toone or more embodiments of the present disclosure.

Referring to FIGS. 4A-4D, in one or more embodiments, the sensing pixels404 of odd-numbered sensing pixel rows 408, 412 of the 3D depth sensingregion may have a first rectangular-shape arrangement (indicated by theeleventh interval 414 and twelfth interval 416 being different from eachother) and the sensing pixels 404 of even-numbered sensing pixels rows410 of the 3D depth sensing region may have a second rectangular-shapearrangement. The first rectangular-shape arrangement may be offset fromthe second rectangular-shape arrangement in the row direction (e.g.,first direction DR1) and/or the column direction (e.g., second directionDR2) at different intervals (e.g., thirteenth interval 418 beingdifferent from the fourteenth interval 420). In other words, the sensingpixels of the 3D depth sensing region may have a zig-zag-shapearrangement. However, the present disclosure is not limited thereto, andany suitable shape arrangement may be used. For example, in one or moreembodiments, a diamond, square, or rectangular shape arrangement may beused.

In the embodiments shown in FIG. 4A-4B, some sensing pixel columns ofthe 2D sensing region may not be interrupted (or displaced) by a sensingpixel 404 of the 3D depth sensing region. Further, each sensing pixel404 of the 3D depth sensing region may be separated from another sensingpixel 404 of the 3D depth sensing region by nine sensing pixels 401,402, 403 or 402, 405 in the row direction (e.g., first direction DR1)and seventeen sensing pixels 401, 402, 403 or 402, 405 in the columndirection (e.g., second direction DR2). However, the present disclosureis not limited thereto. For example, the sensing pixels 404 of the 3Ddepth sensing region may be separated from each other by any suitablenumber of sensing pixels 401, 402, 403 or 402, 405 of the 2D sensingregion in the row direction (e.g., the first direction DR1) and/or thecolumn direction (e.g., the second direction DR2). In the embodimentsshown in FIGS. 4C-4D, each sensing pixel 404 of the 3D depth sensingregion may be separated from another sensing pixel 404 of the 3D depthsensing region by nine sensing pixels 401, 402, 403 or 402, 405 in therow direction (e.g., first direction DR1) and fifteen sensing pixels401, 402, 403 or 402, 405 in the column direction (e.g., seconddirection DR2).

Although the sensing pixels 404 of the 3D depth sensing region shown inFIGS. 4A-4B have an area of a particular size and shape (e.g., a squareshape area with an area ratio of one sensing pixel 404 of the 3D depthsensing region to nine sensing pixels 401, 402, 403 or 402, 405 of the2D sensing region), the sensing pixels 404 of the 3D depth sensingregion may be any suitable size or shape. For example, as shown in FIGS.4C-4D, the sensing pixels 404 of the 3D depth sensing region may have alarger area and that area may be rectangular (e.g., a rectangular shapearea with an area ratio of one sensing pixel 404 of the 3D depth sensingregion to 15 sensing pixels 401, 402, 403 or 402, 405 of the 2D sensingregion). In the case of a sensing pixel 404 of the 3D depth sensingregion having a rectangular shape, the sensing pixel 404 of the 3D depthsensing region may be more suitable for detecting narrow objectscompared to a sensing pixel 404 of the 3D depth sensing region having asquare shape. For example, a square shape may be more suitable fordetecting an object which has a square-shape aspect ratio and arectangular shape may more suitable for detecting a narrow object with arectangular-shape aspect ratio. Further, a larger area for a sensingpixel 404 of the 3D depth sensing region may be more appropriate fordetecting objects at greater distances (e.g., NIR reflected from theobject may be more likely to be directed (or focused) onto a portion ofthe sensing pixel 404 of the 3D depth sensing region). By positioninglarger area sensing pixels 404 of the 3D depth sensing region adjacentto each other, a sensing signal provided by each sensing pixel 404 ofthe 3D depth sensing region may be confirmed by adjacent sensing pixels404 of the 3D depth sensing region to avoid false-positive 3D depthinformation due to noise and the like at a single sensing pixel 404 ofthe 3D depth sensing region. Therefore, depending on the application, itmay be desirable to have sensing pixels of the 3D depth sensing regionof suitable sizes and shapes.

In one or more embodiments, as shown in FIGS. 4A-4D, a plurality ofmetalenses (e.g., metalens 406) may cover each sensing pixel 404 of the3D depth sensing region and one or more sensing pixels 401, 402, 403 or402, 405 of the 2D sensing region that may be adjacent to the sensingpixel 404 of the 3D depth sensing region. In this case, light incidenton the metalens 406 may be directed according to wavelength to eitherthe sensing pixels 401, 402, 403 or 402, 405 of the 2D sensing region orthe sensing pixels 404 of the 3D depth sensing region. For example,visible light incident on the metalens 406 may be directed (or focused)on the sensing pixels 401, 402, 403 or 402, 405 of the 2D sensingregion, and NIR light incident on the metalens 406 may be directed (orfocused) on the sensing pixels 404 of the 3D depth sensing regionadjacent (e.g., directly adjacent) to the sensing pixels 401, 402, 403or 402, 405 of the 2D depth sensing region.

In one or more embodiments, the metalens 406 directs (or focuses) lightbased on nanostructures which may be suitably modified to direct (orfocus) set wavelength spectrums of incident light toward respectivesensing regions of the hybrid sensing array as will be described in moredetail with reference to FIGS. 5A-6B below. In this case, a centerportion of the metalens 406 may be designed to direct (or focus)different wavelengths of light from an outer portion of the metalens 406surrounding the center portion of the metalens 406.

Although in the illustrated embodiment a plurality of metalens 406 eachcover a single sensing pixel 404 of the 3D depth sensing region andsensing pixels 401, 402, 403 or 402, 405 of the 2D depth sensing regionadjacent to the sensing pixel 404 of the 3D depth sensing region, thepresent disclosure is not limited thereto. For example, the metalens 406may cover only sensing pixels 401, 402, 403 or 402, 405 of the 2Dsensing region and not the sensing pixel 404 of the 3D depth sensingregion such that the metalens 406 directs (or focuses) visible lighttoward the sensing pixels 401, 402, 403 or 402, 405 of the 2D sensingregion and NIR light toward the sensing pixel 404 of the 3D depthsensing region, and, in other embodiments, the metalens 406 may coveronly sensing pixels 404 of the 3D depth sensing region and not thesensing pixels 401, 402, 403 or 402, 405 of the 2D sensing region suchthat the metalens 406 directs (or focuses) visible light toward thesensing pixels of the 2D sensing region and NIR light toward the SPADs.Accordingly, the metalens 406 may be any shape and cover any number ortype of sensing pixels of the hybrid sensing array with suitableadjustments to the metalens 406.

Although FIGS. 4A-4D illustrate a plurality of metalenses, the presentdisclosure is not limited thereto. For example, any number of metalenses406 may be used to cover the hybrid sensing array, for example, a globallens (i.e., a single metalens) having nanostructures thereon may be usedto cover the entire hybrid sensing array with suitable changes to thenanostructures of the global lens. Although the embodiment of FIGS. 4Aand 4C include red, green, and blue sensing pixels, the presentdisclosure is not limited thereto. For example, as shown in FIGS. 4B and4D, the sensing pixels may include red and clear sensing pixels insteadof red, green, and blue sensing pixels.

FIGS. 5A-5C are a cross-sectional views 500 a, 500 b, 500 c of a hybridsensing array with a metalens 502 and a microlens 504 on the hybridsensing array, according to one or more embodiments of the presentdisclosure.

Referring to FIGS. 5A-5C, according to one or more example embodimentsof the present disclosure, a microlens 504 and a metalens 502 may be ona hybrid sensing array 508. The metalens 502 may include a plurality ofnanostructures 512, 514, 518, 522, 524, 526 (e.g., a plurality of thindielectric nano-antenna blocks or scatterrers) disposed on rigid or aflexible transparent substrate suitable for transmitting one or moretarget wavelength spectrums of external light therethrough. The metalens502 (e.g., the flexible transparent substrate of the metalens 502) maybe any suitable shape such as, for example, a circular, cylindrical,rectangular, or square shape. In one or more embodiments, the metalens502 may have polarization-independent properties.

The shape, size, and position of each of the nanostructures 512, 514,518, 522, 524, 526 may affect properties of light such as phase,polarization, and focal point. Therefore, each of the nanostructures512, 514, 518, 522, 524, 526 may be disposed on the substrate to havedifferent geometric structures and/or arrangements to direct (or focus)different target wavelength spectrums of external light towardsdifferent portions of the hybrid sensing array 508. In one or moreembodiments, the nanostructures 512, 514, 518, 522, 524, 526 may becircular, cylindrical, rectangular, square, etc., in shape, and thenanostructures 512, 514, 518, 522, 524, 526 may be arranged such thatthe one or more metalenses (e.g., metalens 502) exhibitpolarization-independent properties.

In one or more embodiments, as shown in FIG. 5A, the metalens 502includes a first group 510 of nanostructures (e.g., a firstnanostructure 512 and a second nanostructure 514) overlapping (e.g.,overlapping in the third direction DR3) a first sensing pixel 511 of the2D sensing region and a second group 516 of nanostructures (e.g., athird nanostructure 518) overlapping (e.g., overlapping in the thirddirection DR3) a second sensing pixel 517 of the 2D sensing region, anda third group 520 of nanostructures (e.g., a fourth nanostructure 522, afifth nanostructure 524, and a sixth nanostructure 526) overlapping(e.g., overlapping in the third direction DR3) a sensing pixel 519 ofthe 3D depth sensing region.

In one or more embodiments, the first group 510 of nanostructures (e.g.,the first nanostructure 512 and the second nanostructure 514) may beconfigured to direct (or focus) different wavelength spectrums ofexternal light to different portions of the hybrid sensing array 508.For example, the first nanostructure 512 overlapping the first sensingpixel 511 may be configured to direct (or focus) light in the NIRspectrum (e.g., greater than 0.8 micrometers) toward the sensing pixel519 of the 3D depth sensing region directly adjacent to the firstsensing pixel 511 of the 2D sensing region, and the second nanostructure514 overlapping the first sensing pixel 511 of the 2D sensing region maybe configured to direct (or focus) light in the visible to NIR spectrum(e.g., about 0.4 micrometers to about 0.8 micrometers) toward the firstsensing pixel 511 of the 2D sensing region. Accordingly, the first group510 of nanostructures may function as a beam splitter by directing (orfocusing) different wavelength spectrums towards different sensingregions of the hybrid sensing array.

In one or more embodiments, each of the nanostructures (e.g., the thirdnanostructure 518) of the second group 516 may be configured to direct(or focus) the same wavelength spectrum of external light to the sameportion of the hybrid sensing array 508. For example, each of thenanostructures (e.g., the third nanostructure 518) of the second group516 may be configured to direct (or focus) light in the visible to NIRspectrum (e.g., about 0.4 micrometers to about 0.8 micrometers) towardthe second sensing pixel 517 of the 2D sensing region. In other words,none of the nanostructures in the second group 516 overlapping (e.g.,overlapping in the third direction DR3) the second sensing pixel 517 ofthe 2D sensing region direct (or focus) light toward adjacent sensingpixels. However, the present disclosure is not limited thereto. Forexample, the second group 516 may have nanostructures directing (orfocusing) light toward adjacent and/or directly adjacent sensing pixelsto function as a beam splitter.

In one or more embodiments, the third group 520 of nanostructures (e.g.,the fourth nanostructure 522, the fifth nanostructure 524, and the sixthnanostructure 526) may be configured to direct (or focus) differentwavelength spectrums of external light to different portions of thehybrid sensing array 508. For example, the fourth nanostructure 522overlapping the sensing pixel 519 of the 3D depth sensing region may beconfigured to direct (or focus) light in the NIR spectrum (e.g., greaterthan 0.8 micrometers) toward the sensing pixel 519 of the 3D depthsensing region, the fifth nanostructure 524 overlapping the sensingpixel 519 of the 3D depth may be configured to direct (or focus) lightin the visible to NIR spectrum (e.g., about 0.4 micrometers to about 0.8micrometers) toward the first sensing pixel 511 of the 2D sensing regiondirectly adjacent to the sensing pixel 519 of the 3D depth sensingregion, and the sixth nanostructure 526 overlapping the sensing pixel519 of the 3D depth may be configured to direct (or focus) light in thevisible to NIR spectrum (e.g., about 0.4 micrometers to about 0.8micrometers) toward the third sensing pixel 521 of the 2D sensing regiondirectly adjacent to the sensing pixel 519 of the 3D depth sensingregion. Accordingly, the third group 520 of nanostructures may functionas a beam splitter by directing (or focusing) different wavelengthspectrums towards different sensing regions of the hybrid sensing array.In this case, the third group 520 may direct (or focus) light towardsthree different sensing pixels. However, the present disclosure is notlimited thereto, and any suitable directing (or focusing) arrangement ofnanostructures may be used.

Although the metalens is shown and described in FIG. 5A as includingnanostructures overlapping each of the sensing pixels, the presentdisclosure is not limited thereto. For example, as shown in FIG. 5B,some sensing pixels (e.g., the sensing pixel 519 of the 3D depth sensingregion in the embodiment of FIG. 5B) may not have nanostructuresoverlapping the sensing pixel 519 of the 3D depth sensing region in thethird direction.

Although the metalens is shown and described in FIG. 5A as includingnanostructures overlapping sensing pixels where beam splitting involvesdirectly adjacent sensing pixels, the present disclosure is not limitedthereto. For example, as shown in FIG. 5B, the second group 516 mayinclude a seventh nanostructure 528 in addition to the thirdnanostructure 518 shown in the embodiment FIG. 5A. As shown in FIG. 5B,the third nanostructure 518 of the second group 516 may be configured todirect (or focus) light in the visible to NIR spectrum (e.g., about 0.4micrometers to about 0.8 micrometers) toward the second sensing pixel517 of the 2D sensing region, and the seventh nanostructure 528 of thesecond group 516 may be configured to direct (or focus) light in the NIRspectrum (e.g., greater than 0.8 micrometers) toward the sensing pixel519 of the 3D depth sensing region adjacent to the second sensing pixel517 of the 2D sensing region with intervening first sensing pixel 511 ofthe 2D sensing region between the second sensing pixel 517 of the 2Dsensing region and the sensing pixel 519 of the 3D depth sensing region.

In one or more embodiments, one or more microlenses (e.g., microlens504) may help focus incident light toward target portions of the sensingpixels. The microlens 504 may be position below the metalens 502 asshown in the embodiments of FIGS. 5A and 5B, and in other embodiments,the microlens 504 may be positioned above the metalens 502 as shown inembodiments FIG. 5C. In other words, the microlens 504 may be betweenthe metalens 502 and the hybrid sensing array in the embodiments ofFIGS. 5A and 5B, and in other embodiments, the metalens 502 may bebetween the microlens 504 and the hybrid sensing array as shown in theembodiment of FIG. 5C.

In one or more embodiments, a plurality of color filters 506 may filterset wavelength spectrums of light. Each of the color filters 506 in theembodiments of FIGS. 5A-5C correspond to the wavelength of light sensedby the sensing pixel overlapping the color filters 506 in the thicknessdirection (e.g., the third direction DR3). For example, a blue light maytransmit through a blue color filter of the plurality of color filters506 overlapping a blue sensing pixel (e.g., the first sensing pixel 511)and a NIR light may transmit through a NIR color filter of the pluralityof color filters 506 overlapping the sensing pixel 519.

Accordingly, as shown in FIGS. 5A-5C, external light incident on themetalens 502 including one or more nanostructures (e.g., structures forscattering lobes of different wavelength light) may be directed (orfocused) based on one or more target wavelength spectrums toward the 2Dsensing region and/or the 3D depth sensing region of the hybrid sensingarray 508.

Although a microlens 504 is shown in FIGS. 5A-5C, the microlens 504provides dedicated light focusing capabilities to supplement thecapability of the metalens 502 to focus and redirect target wavelengthspectrums. Therefore, in one or more embodiments, a microlens 504 maynot be present where additional light focusing may not necessary.However, in the embodiments of FIGS. 5A-5C, the microlens 504 may bepresent to help resolve issues with the chief ray angle (“CRA”) or FOVof the 2D sensing region and the 3D depth sensing region.

FIG. 6A is a block diagram including a portion of a global metalens 602and an exploded view of the portion of the global metalens 602 on ahybrid sensing array, according to one or more embodiments of thepresent disclosure.

In one or more embodiments, external light 600 including NIR light andvisible light may be incident on the global metalens 602. The globalmetalens may include one or more nanostructures 608 directing (orfocusing) different wavelength spectrums of the external light 600 suchthat external light 600 is split into visible light and NIR light. Inone or more embodiments, visible light may be directed (or focused)towards the sensing pixels of the 2D sensing region 604 including PDsand NIR light may be directed (or focused) towards the sensing pixels ofthe 3D depth sensing region 606 including one or more SPADs.

Accordingly the global metalens 602 may cover an entire hybrid sensingarray and different portions of the global metalens 602 may direct (orfocus) visible light to portions of the 2D sensing region proximate tothe portion of the global metalens 602 and the different portions of theglobal metalens 602 may direct (or focus) NIR light to portions of the3D depth sensing region proximate to the portion of the global metalens602.

FIG. 6B is an exploded block diagram including a lens (e.g., a globallens) 610 and a global metalens 612 on a hybrid sensing array 614,according to one or more embodiments of the present disclosure.

Referring to FIG. 6B, in one or more embodiments, a hybrid sensing array614 may include a 2D sensing region 616 and a 3D depth sensing region618 adjacent to each other in a side-by-side arrangement with the 2Dsensing region 616 and the 3D depth sensing region 618 sharing the sameFOV because of one main lens. In one or more embodiments, the globalmetalens 612 may be centered to cover the 2D sensing region 616 suchthat visible light directed (or focused) by the global metalens 612 ontothe red, green, and blue sensing pixels of the 2D sensing region resultsin high 2D image quality. Further, the global metalens 612 directs (orfocuses) sufficient NIR light towards the 3D depth sensing region 618 toprovide 3D depth information. Accordingly, centering the global metalens612 to cover the 2D sensing region 616 assists in providing ahigh-resolution 2D image while redirected (or focused) NIR light to theadjacent 3D depth sensing region 618 provides sufficient NIR light forthe sensing pixels of the 3D depth sensing region 618 to detect the NIRlight and provide 3D depth information.

FIG. 7A is a plan view and a cross-sectional view of a hybrid sensor106, according to one or more embodiments of the present disclosure.FIG. 7B is a cross-sectional view of a hybrid sensor 106, according toone or more embodiments of the present disclosure.

Referring to FIGS. 7A-7B, in one or more embodiments, a hybrid sensor106 may include a hybrid sensing array 702, a main lens 708, a globalmetalens 710, and a printed circuit board 712 (“PCB”). The hybridsensing array 702 may be positioned on the PCB 712 and include a 2Dsensing region 704 and a 3D depth sensing region 706 in a side-by-sidearrangement.

As shown in FIG. 7A, a main lens 708 and a global metalens 710 may coveror overlap the hybrid sensing array 702 such that the main lens 708 andthe global metalens 710 are centered over the 2D sensing region 704. Inone or more embodiments, the entire 2D sensing region 704 of the hybridsensing array 702 and the entire 3D depth sensing region of the hybridsensing array may overlap the global metalens 710 and the main lens 708.However, the present disclosure is not limited thereto. For example, asshown in FIG. 7B, the entire 2D sensing region 704 of the hybrid sensingarray 702 may overlap the global metalens 710 and the main lens 708while less than the entire 3D depth sensing region 706 may overlap withthe global metalens 710 and the main lens 708.

In one or more embodiments, the global metalens 710 may be between themain lens 708 and the 2D sensing region 704 such that external lightincident on the main lens 708 passes through the global metalens 710prior to being detected by the hybrid sensing array 702. Light passingthrough the global metalens 710 may be directed (or focused) by one ormore nanostructures 714 positioned on a transparent substrate (e.g., aglass substrate) 716. Each of the one or more nanostructures may direct(or focus) visible light towards the 2D sensing region 704 or direct (orfocus) NIR light towards the 3D depth sensing region 706. Because lightdetected by the 2D sensing region 704 and the 3D depth sensing passesthrough a shared main lens 708, both the 2D sensing region 704 and the3D depth sensing region 706 have the same FOV. Accordingly, centeringthe main lens 708 and the global metalens 710 to cover the 2D sensingregion 704 assists in providing a high-resolution 2D image that may beaugmented with 3D depth information as discussed in more detail below.

As shown in FIG. 7A, the planar area and distances D2, D3, and D4between the main lens 708, global metalens 710, and hybrid sensing array702 may be set such that each sensing pixel of the hybrid sensing array702 may be exposed to directed (or focused) light. The focal point ofthe main lens 708 may be spaced apart from the main lens 708 by adistance D4 and may be spaced apart from the global metalens 710 by adistance D3. In one or more embodiments, the focal point of the mainlens may be between the main lens 708 and the global metalens 710.Further, the hybrid sensing array 702 may be spaced apart from theglobal metalens 710 by a distance D2 such that each of the sensingpixels of the hybrid sensing array 702 may detect directed (or focused)light from the main lens 708 and/or the global metalens 710.

Accordingly, a compact, monolithic hybrid sensor 106 is provided with amain lens 708 and a global metalens 710 that covers or overlaps thehybrid sensing array 702 to provide 2D image data and 3D depthinformation from a shared FOV. Further, the side-by-side arrangement ofthe 2D sensing region 704 and the 3D depth sensing region 706 assists inproviding a high resolution 2D image because sensing pixels of the 3Ddepth sensing region 706 do not interrupt (or displace) sensing pixelsof the 2D sensing region 704 leading to “missing” sensing pixels of the2D sensing region 704.

FIG. 8 is a flow chart of a method 800 for object detection andtracking, according to one or more embodiments of the presentdisclosure.

Referring to FIG. 8, an object detector 212 of the processing circuit204 may use a neural-network based object detection system to detect andtrack objects in 2D image data based on data from the 2D sensing region.

In one or more embodiments, the object detector 212 may detect an objectusing 2D image data (802). The object detector 212 may determine whetherthe object detected is a new object (804). For example, the objectdetector 212 may determine whether the object is a new object based onwhether the object detected in the 2D image data is already stored in anobject table 810.

If the object is an object that was previously or already detected, thenthe object detector 212 continues to detect other objects in the imageas part of the object detection loop. If the object is a new object thatwas not previously or already detected in the current 2D image data,then the object detector 212 works with the ROI controller 210 of theprocessing circuit 204 to activate the 3D depth sensing region. Forexample, the object detector 212 may detect an object (e.g., apedestrian) using the 2D image data (802), and identify coordinates ofthat object (e.g., the pedestrian) in the 2D image data. Based on thecoordinates of the object in the 2D image data, the ROI controller 210may activate portions of the light source 104 (e.g., portions of anarray of VCSELs) to illuminate the object (e.g., the pedestrian) in thereal-world. In one or more embodiments, the ROI controller 210 may sendsignals to the sensing circuit 202 to select specific sensing pixels(e.g., one or more sensing pixels including SPADs) of the 3D depthsensing region according to the coordinates of the object identifiedfrom the 2D image. In one or more embodiments, the sensing pixels of the3D depth sensing region driven by the sensing circuit 202 may beadjacent or proximate to the sensing pixels of the 2D sensing regionthat provide sensing signals corresponding to the 2D image data of theobject (e.g., the pedestrian).

Accordingly, 3D depth sensing information may be collected by the hybridsensor 106 (i.e., 3D depth sensing is turned on (806)). By selectivelyturning on 3D depth sensing in response to a new object, battery powerusage may be reduced and less computation resources may be used. Basedon a combination of the 2D image data and the 3D depth information, theobject detector 212 may estimate information regarding the new object(808) such as shape, identity, and/or 2D size information according to2D sensing information and depth according to 3D depth sensinginformation, and store the estimated information in the object table 810before returning to detecting other objects (802) thereby completing adetection loop. In one or more embodiments, the object table 810 mayindicate whether the object is an OOI which should be tracked.

Accordingly, in one or more embodiments, the object detector 212 maytrack an object identified by the object table based on the detectionloop. For example, for ADAS or autonomous vehicle applications, a treeand a pedestrian may be identified by the detection loop. In this case,the object detector 212 may determine that a tree in the backgroundshould not be tracked. On the other hand, the object detector 212 maydetermine that a pedestrian should be tracked (e.g., because apedestrian may move). Accordingly, the pedestrian identified in the 2Dimage data may be tracked as an OOI using ROI sensing. Although theexample of a tree and pedestrian is provided, the present disclosure isnot limited thereto. Depending on the application, any object may bedesignated for tracking based on several factors (e.g., location,importance, distance, mobility, and the like).

In one or more embodiments, the object detector 212 may continuallytrack (e.g., track the location of) the object in the 2D image data(812). If the object detector 212 loses track of the tracked object(e.g., the tracked object exits the FOV or an error occurs) in the 2Dimage data (814), then the object tracking ends until the detection loopdetects another object or the same object again to track (i.e., theobject detector 212 exits the tracking loop and returns to the detectionloop). Therefore, in the detection loop, there is a possibility that thetracked object which was lost will be reidentified and tracked again tocorrect any possible errors.

If the object detector 212 is tracking the object (e.g., an OOI) and acritical condition is encountered (e.g., a condition where the OOI is infront in an ADAS application) (816), the object detector 212 may workwith the ROI controller 210 of the processing circuit 204 to activatethe 3D depth sensing region. The ROI controller 210 may send signals tothe sensing circuit 202 to select specific sensing pixels of the 3Ddepth sensing region and/or activate portions of the light source 104(e.g., portions of an array of VCSELs) to illuminate the detected objectin the real-world. Accordingly, 3D depth sensing information may becollected by the hybrid sensor 106 (i.e., 3D depth sensing is turned on(818)). By selectively turning on 3D depth sensing (818) in response toa critical condition, battery power usage may be reduced and lesscomputational resources may be used. Based on new information from the3D depth sensing, the object detector may augment or update the 3D depthinformation for the tracked object (820). In one or more embodiments,the combination of the 2D image data and the 3D depth information mayalso enable the object detector to reassess or update object information(820) such as shape, identity, and/or 2D size information according to2D sensing information and depth according to 3D depth sensinginformation in the object table before returning to tracking the objectuntil the tracked object is lost (814) or another critical condition ismet (816).

Although the FIG. 8 is described with reference to a detection loop anda tracking loop for a single object, the present disclosure is notlimited thereto. For example, a plurality of objects may be detected andtracked concurrently as desired. Further, the detection loop may bealways active while the tracking loop may be active as objects may bedesignated by the object detector 212 for tracking (e.g., designated asan OOI). In one or more embodiments, all of the objects detected by theobject detector 212 may be tracked and, in other embodiments, less thanall of the objects detected by the object detector 212 may be tracked asdesired.

FIG. 9 is a flow chart 900 of a method for providing a high resolution2D image and a high resolution 3D image, according to one or moreembodiments of the present disclosure.

Referring to FIG. 9, according to one or more example embodiments of thepresent disclosure, an image signal processor 214 may provide orreconstruct 2D and/or 3D images using data from the sensing circuit 202.

In one or more embodiments, the image signal processor 214 may provideor reconstruct a 2D image based on the arrangement of the sensing pixelsof the 2D sensing region and the sensing pixels of the 3D depth sensingregion. For example, when sensing pixels of the 2D sensing region areinterwoven with sensing pixels of the 3D depth sensing region, the 2Dimage data may be missing patches of 2D image data because the sensingpixel of the 3D depth sensing region occupies a region of the hybridsensing array that interrupts (or displaces) sensing pixel rows and/orsensing pixel columns of the 2D sensing region. Therefore, 2D image datagenerating by the 2D sensing region has missing pixels (e.g., one ormore missing green pixels, one or more missing red pixels, and/or one ormore missing blue pixels). However, the present disclosure is notlimited thereto. For example, in one or more embodiments, any sensingpixels interrupted (or displaced) by a sensing pixel of the 3D depthsensing region may result in “missing” pixels, and therefore, missingpixels can be of any color and/or type depending on the sensing pixelsof the 2D sensing region that may be interrupted (or displaced) by asensing pixel of the 3D depth sensing region.

In the illustrated embodiment, in response to 2D image data from the 2Dsensing region, the image signal processor 214 may interpolate missinggreen pixels (902), and interpolate missing red and blue pixels (904).In response to interpolating the missing pixels, the image signalprocessor 214 may complete a Bayer image (906). In response tocompleting the Bayer image, the image signal processor 214 may use ademosaicing algorithm to reconstruct a full color image (908).Accordingly, a full color image may be reconstructed for use in variousapplications (910) (e.g., use by the object detector 212 of theprocessing circuit 204).

Although reconstruction of the 2D image is described with reference tointerpolating missing pixels (902, 904), depending on the embodiment ofthe present disclosure, interpolating missing pixels may be skipped(902, 904). For example, in the case where the sensing pixels of the 2Dsensing region and the sensing pixels of the 3D depth sensing region arein a side-by-side arrangement as shown in the embodiments of FIGS. 2B,3I, 6B, and 7A-7B, there may be no missing pixels to interpolate, andtherefore, interpolating missing pixels may be skipped. In other words,the embodiments of FIGS. 2B, 3I, 6B, and 7A-7B may start from step 910.

In one or more embodiments, in response to reconstructing a full colorimage, the image signal processor 214 may interpolate missing depthpixels by weighted averaging (912). In other words, 3D depth informationfrom the 3D depth sensing region may not have the same resolution as the2D image data, and therefore, missing 3D depth information may bedetermined based on interpolation using sensed 3D depth information andweighted averaging. Accordingly, a high resolution 3D image may beprovided including both the high resolution 2D image as a foundation orbase and 3D depth information augmenting or updating the high resolution2D image.

Accordingly, as disclosed herein, embodiments of the present disclosureprovide high resolution 2D and 3D images based on a hybrid sensing arrayincluding a 2D sensing region and a 3D depth sensing region with ashared FOV. Further, systems and methods of one or more embodiments ofthe present disclosure provide ROI sensing capabilities based on objectdetection and tracking algorithms to save on power and computationalresources.

In the drawings, the relative sizes of elements, layers, and regions maybe exaggerated and/or simplified for clarity.

It will be understood that, although the terms “first,” “second,”“third,” etc., may be used herein to describe various elements,components, regions, layers and/or sections, these elements, components,regions, layers and/or sections should not be limited by these terms.These terms are used to distinguish one element, component, region,layer or section from another element, component, region, layer orsection. Thus, a first element, component, region, layer or sectiondescribed below could be termed a second element, component, region,layer or section, without departing from the spirit and scope of thepresent disclosure.

It will be understood that when an element or layer is referred to asbeing “on,” “connected to,” or “coupled to” another element or layer, itcan be directly on, connected to, or coupled to the other element orlayer, or one or more intervening elements or layers may be present. Inaddition, it will also be understood that when an element or layer isreferred to as being “between” two elements or layers, it can be theonly element or layer between the two elements or layers, or one or moreintervening elements or layers may also be present.

The terminology used herein is for the purpose of describing particularembodiments and is not intended to be limiting of the presentdisclosure. As used herein, the singular forms “a” and “an” are intendedto include the plural forms as well, unless the context clearlyindicates otherwise. It will be further understood that the terms“comprises,” “comprising,” “includes,” and “including,” “has,” “have,”and “having,” when used in this specification, specify the presence ofthe stated features, integers, steps, operations, elements, and/orcomponents, but do not preclude the presence or addition of one or moreother features, integers, steps, operations, elements, components,and/or groups thereof. As used herein, the term “and/or” includes anyand all combinations of one or more of the associated listed items.Expressions such as “at least one of,” when preceding a list ofelements, modify the entire list of elements and do not modify theindividual elements of the list.

Further, the use of “may” when describing embodiments of the presentdisclosure refers to “one or more embodiments of the presentdisclosure.” As used herein, the terms “use,” “using,” and “used” may beconsidered synonymous with the terms “utilize,” “utilizing,” and“utilized,” respectively.

As used herein, the terms “substantially”, “about”, and similar termsare used as terms of approximation and not as terms of degree, and areintended to account for the inherent deviations in measured orcalculated values that would be recognized by those of ordinary skill inthe art.

Any numerical range recited herein is intended to include all sub-rangesof the same numerical precision subsumed within the recited range. Forexample, a range of “1.0 to 10.0” is intended to include all subrangesbetween (and including) the recited minimum value of 1.0 and the recitedmaximum value of 10.0, that is, having a minimum value equal to orgreater than 1.0 and a maximum value equal to or less than 10.0, suchas, for example, 2.4 to 7.6. Any maximum numerical limitation recitedherein is intended to include all lower numerical limitations subsumedtherein and any minimum numerical limitation recited in thisspecification is intended to include all higher numerical limitationssubsumed therein. Accordingly, Applicant reserves the right to amendthis specification, including the claims, to expressly recite anysub-range subsumed within the ranges expressly recited herein.

Unless otherwise defined, all terms (including technical and scientificterms) used herein have the same meaning as commonly understood by oneof ordinary skill in the art to which the present disclosure belongs. Itwill be further understood that terms, such as those defined in commonlyused dictionaries, should be interpreted as having a meaning that isconsistent with their meaning in the context of the relevant art and/orthe present specification, and should not be interpreted in an idealizedor overly formal sense, unless expressly so defined herein.

Although some example embodiments have been described, those skilled inthe art will readily appreciate that various modifications are possiblein the example embodiments without departing from the spirit and scopeof the present disclosure. It will be understood that descriptions offeatures or aspects within each embodiment should typically beconsidered as available for other similar features or aspects in otherembodiments, unless otherwise described. Therefore, it is to beunderstood that the foregoing is illustrative of various exampleembodiments and is not to be construed as limited to the specificexample embodiments disclosed herein, and that various modifications tothe disclosed example embodiments, as well as other example embodiments,are intended to be included within the spirit and scope of the presentdisclosure as defined in the appended claims, and their equivalents.

What is claimed is:
 1. A 3D sensing system comprising: a light sourceconfigured to emit light; a hybrid sensing array comprising a 2D sensingregion configured to detect ambient light reflected from an object and a3D depth sensing region configured to detect the light emitted by thelight source and reflected from the object; a metalens on the hybridsensing array, the metalens comprising a first nanostructure and asecond nanostructure, the first nanostructure being configured to directthe ambient light reflected from the object towards the 2D sensingregion, and the second nanostructure being configured to direct thelight emitted by the light source and reflected from the object towardsthe 3D depth sensing region; and a processing circuit configured tocombine 2D image information provided by the 2D sensing region and 3Dinformation provided by the 3D depth sensing region to generate acombined 3D image.
 2. The 3D sensing system of claim 1, wherein theprocessing circuit is further configured to: determine 2D imageinformation of the object based on the 2D sensing region; determine 3Ddepth information of the object based on the 3D depth sensing region;and update the 2D image information of the object with the 3D depthinformation.
 3. The 3D sensing system of claim 1, further comprising arow decoder and a column amplifier, the row decoder and the columnamplifier being stacked below the hybrid sensing array.
 4. The 3Dsensing system of claim 1, wherein the metalens is circular,cylindrical, rectangular, or square in shape and ispolarization-independent.
 5. The 3D sensing system of claim 1, whereinthe 2D sensing region comprises one or more photodiodes and the 3D depthsensing region comprises one or more APDs or SPADs.
 6. The 3D sensingsystem of claim 1, wherein the metalens is configured to direct NIRlight to the 3D depth sensing region.
 7. The 3D sensing system of claim1, wherein the processing circuit is further configured to: detect theobject based on the 2D image information; generate a region of interestbased on coordinates of the object in the 2D image information; andilluminate the region of interest using the light source.
 8. The 3Dsensing system of claim 7, wherein the processing circuit is furtherconfigured to: determine an identity of the object based on the 2D imageinformation and 3D depth information; track the object based on 2D imageinformation; and update the 2D image information with 3D depthinformation in response to a critical condition.
 9. A method ofproviding an image based on a hybrid sensing array, the methodcomprising: emitting light from a light source; detecting, by a 2Dsensing region of the hybrid sensing array, ambient light reflected froman object; detecting, by a 3D depth sensing region of the hybrid sensingarray, the light emitted by the light source and reflected from theobject towards the 3D depth sensing region; directing, by a firstnanostructure of a metalens on the hybrid sensing array, ambient lighttowards the 2D sensing region; directing, by a second nanostructure ofthe metalens of the hybrid sensing array, light emitted by the lightsource and reflected from the object towards the 3D depth sensingregion; and combining, by a processing circuit, 2D image informationprovided by the 2D sensing region and 3D information provided by the 3Ddepth sensing region to generate a combined 3D image.
 10. The method ofclaim 9, the method further comprising determining, by the processingcircuit, 2D image information of the object based on the 2D sensingregion; determining, by the processing circuit, 3D depth information ofthe object based on the 3D depth sensing region; and updating, by theprocessing circuit, the 2D image information of the object with the 3Ddepth information.
 11. The method of claim 9, wherein the hybrid sensingarray detecting the ambient light and the light emitted by the lightsource is stacked above a row decoder and a column amplifier.
 12. Themethod of claim 9, wherein the metalens directing ambient light towardsthe 2D sensing region and directing light emitted by the light sourceand reflected from the object towards the 3D depth sensing region iscircular, cylindrical, rectangular, or square in shape and ispolarization-independent.
 13. The method of claim 9, wherein the 2Dsensing region comprises one or more photodiodes to detect ambient lightand the 3D depth sensing region comprises APDs or SPADs to detect thelight emitted by the light source and reflected off the object.
 14. Themethod of claim 9, wherein the directing, by the metalens on the hybridsensing array, ambient light towards the 2D sensing region comprisesdirecting visible light towards the 2D sensing region, and wherein thedirecting, by the metalens of the hybrid sensing array, light emitted bythe light source and reflected from the object towards the 3D depthsensing region comprises directing NIR light towards the 3D depthsensing region.
 15. The method of claim 9, the method further comprisingdetecting, by the processing circuit, the object based on the 2D imageinformation; generating, by the processing circuit, a region of interestbased on coordinates of the object in the 2D image information; andilluminating, by the processing circuit, the region of interest usingthe light source.
 16. The method of claim 9, the method furthercomprising determining, by the processing circuit, an identity of theobject based on the 2D image information and 3D depth information;tracking, by the processing circuit, the object based on 2D imageinformation; and updating, by the processing circuit, the 2D imageinformation with 3D depth information in response to a criticalcondition.
 17. A 3D sensing system comprising: a light source configuredto emit light; a 2D sensing region to detect visible light reflectedfrom an object; a 3D depth sensing region to detect near infrared lightreflected from the object; a metalens covering the 2D sensing region,the metalens comprising a first nanostructure and a secondnanostructure, the first nanostructure being configured to directvisible light reflected from the object toward the 2D sensing region,and the second nanostructure being configured to direct the lightemitted by the light source and reflected from the object towards the 3Ddepth sensing region; and a processing circuit configured to: determine2D image information of the object based on the 2D sensing region;determine 3D depth information of the object based on the 3D depthsensing region; and update the 2D image information of the object withthe 3D depth information.
 18. The 3D sensing system of claim 17, furthercomprising a row decoder and a column amplifier, the row decoder and thecolumn amplifier being stacked below the 2D sensing region and the 3Ddepth sensing region.
 19. The 3D sensing system of claim 17, wherein themetalens is circular, cylindrical, rectangular, or square in shape andis polarization-independent.
 20. The 3D sensing system of claim 17,wherein the 2D sensing region comprises one or more photodiodes and the3D depth sensing region comprises one or more SPADs or APDs.